TITLE OF THE INVENTION 

A METHOD AND AN APPARATUS TO DIVIDE IMAGE BLOCKS 
CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] This application claims priority from Korean Patent Application No. 2003-07118, filed 
February 5, 2003, the contents of which are incorporated herein in their entirety by reference. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 
[0002] The present invention relates to a method and apparatus to divide image blocks, and 
more particularly, to a method and apparatus to improve the quality of intermediate images. To 
achieve such improvements, this invention proposes a new standard to divide image blocks 
that can prevent flickering in synthesized intermediate images when splitting image blocks using 
quadtree disparity estimation. The image block splitting is followed by disparity estimation 
required for synthesis of intermediate views used to represent a three dimensional (3D) image. 

2. Description of the Related Art 

[0003] To realize an imaging and communication system that provides a high degree of 
realism and naturalness, it is necessary to develop 3D image processing technology that can 
naturally represent images according to human visual characteristics. 3D image processing 
employs binocular parallax, which is a difference in depth of an object perceived by left and right 
eyes. Processing and transmission of binocular images is of great concern in the field of next 
generation visual communication. 

[0004] However, one significant problem associated with such image processing and 
transmission is a large amount of information contained in stereoscopic images, considering the 
fact that most images are color or moving images as well as transmission rate over a 
transmission line and processing rate of a transmission (or image processing) system. To 
overcome this problem, it is necessary to research a technique to efficiently and easily 
compress a large amount of information while maintaining the quality of a stereoscopic image. 

[0005] Research is being conducted to develop a method that involves estimating variation in 
objects within an image using the fact that left and right views have high correlation, instead of 
independently encoding the two views, transmitting the variation information and either the left 
or right view, and compensating and restoring the transmitted view to binocular images at a 
receiving terminal. 
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[0006] Further, when the viewpoint of an observer moves or there are several observers, 
multi-view images are needed to create a natural stereoscopic image. However, since 
independent transmission of all multi-view images excessively increases the amount of 
information, reconstruction of multi-view images from binocular images restored at the receiving 
terminal of a binocular image transmission system, often called intermediate view reconstruction 
(IVR) or intermediate view synthesis, is used. In this case, reconstruction can be performed 
using intermediate view interpolation or extrapolation by obtaining variation information related 
to intermediate views from information on variation between the binocular images. 

[0007] 3D images are compressed and decompressed using an MPEG technique applied to 
two dimensional (2D) images. In particular, compression, transmission, and decompression of 
3D images for digital broadcasting are performed using MPEG-2, which is a standard for digital 
broadcasting. 

[0008] As is widely known in the art, MPEG-2 uses block-based compression schemes to 
compress 2D images. Currently, these schemes are also applied to compression of 3D images 
and are known as the most efficient method of 3D image compression. 

[0009] Block-based compression is performed in blocks of a fixed size (for example, 16x16 
pixels), each of which is called a macro block. Compression is achieved by motion estimation in 
units of a macro block and calculating a motion vector, which is the resulting value of estimation, 
and prediction error. When macro blocks of a fixed size are used in synthesizing intermediate 
views from binocular images, as in 2D images for stereoscopic depth perception by a viewer of 
3D images as mentioned above, degradation in the quality of intermediate views may occur. In 
particular, quality degradation due to blurring of the edges of an image becomes a significant 
problem. 

[0010] To overcome this problem, various quadtree disparity estimation approaches that can 
prevent quality degradation by splitting a macro block of a fixed size into smaller sub blocks 
near the edge of an image have been proposed (1998 SPIE Paper: Anthony Mancini and 
Janusz Konrad, "Robust Quadtree-based Disparity Estimation for the Reconstruction of 
Intermediate Stereoscopic Images", and IEEE 0-7803-6685-9/01 Paper: D.R.CIewer, "Efficient 
Multiview Image Compression Using Quadtree Disparity Estimation"). 

[0011] According to a quadtree disparity estimation approach, a block matching technique is 
used to calculate a mean absolute difference (MAD) of each macro block in left-eye and right- 
eye views and MADs of four sub blocks into which the macro block is divided. Then, if a ratio 
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Rmadsubi of the maximum sub block MAD to the minimum sub block MAD within a macro block is 
less than a predetermined threshold, disparity between binocular images is estimated in units of 
macro blocks. Conversely, if Rmadsubi is greater than the threshold, disparity is re-estimated for 
each sub block. Since a method of calculating MADs is described in detail in literature including 
the above-cited references, a detailed description will be omitted. 

[0012] FIG. 1A shows a conventional block splitting algorithm using quadtree disparity 
estimation. As mentioned above, a quadtree disparity estimation approach involves splitting an 
NxN macro block into four (N/2)x(N/2) sub blocks and estimating disparity in units of sub blocks 
when the ratio of the maximum sub block MAD to the minimum sub block MAD within the macro 
block is greater than a predetermined threshold. The conventional block splitting process using 
the quadtree disparity estimation approach will now be described in detail with reference to FIG 
1A. 

[0013] If an NxN macro block that is the highest level block in disparity estimation (hereinafter 
called "Large Macro block (LMB) W ) is input (operation S10), block matching between binocular 
images is implemented to estimate disparity between the binocular images and verifies if the 
resulting estimate is correct. The adequacy and correctness of estimation is determined by 
Rmadsubi, which denotes the ratio of the maximum MAD of any sub block (lower level block ) 
within a macro block (higher level block) to the minimum MAD of any sub block within the same 
macro block. As Rmadsubi becomes greater, the disparity difference between regions in a block 
becomes greater, and thus this estimation becomes less accurate. If the resulting estimate is 
verified as the most ideal one, which means there is little disparity difference between regions 
within the block, Rmadsubi approximates 1 , which is the minimum value. 

[0014] Next, in operation S11 , it is determined whether to split the LMB. This is done by 
determining whether Rmadsubi is greater than a first threshold value Rthi, which is the reference 
value used to judge whether to split the LMB. R th i is decided by experiment. 

[0015] If Rmadsubi is less than R m1 , which means disparity present between binocular images 
is low, LMB is not split (operation S111) and disparity is estimated in units of LMBs (operation 
S16). Then, in operation S17 it is decided whether to split the next LMB. 

[0016] If Rmadsubi is greater than R mil the LMB is split into four (N/2)x(N/2) sub blocks 
(hereinafter called "Middle Sub Blocks (MSBs)") (operation S12), and it is determined whether to 
split each MSB (operation S13). For this determination, the MAD of each (N/4) X(N/4) sub block 
(hereinafter called "Small Sub Block (SSB)) within an MSB is calculated. Then, R ma dsub2, the 
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ratio of the maximum SSB MAD to the minimum SSB MAD within an MSB, is calculated and 
compared to a second threshold value R^, which is a reference value to determine whether to 
split the MSB. Like R^, is decided by experiment. 

[0017] If Rmadsub2 is less than Rth2, the MSB is not split (operation S14) and disparity is 
estimated in units of MSBs (operation S16). Then, in step S17 it is determined whether to split 
the next LMB. 

[0018] If Rmadsub2 is greater than R th2l the MSB is split into four (N/4) X(N/4) SSBs (operation 
S15). Then, in operation S17 it is determined whether to split the next LMB. 

[0019] FIG 1B shows image blocks produced in a block splitting procedure using a quadtree 
disparity estimation approach. Each block is contained in a 720x288 image frame. A block 
splitting technique proposed by the present invention is applied to the image blocks shown in 
FIG 1B. For an image created by interlaced scanning, in particular, this splitting may be 
performed in frames or fields. As mentioned above, the conventional block splitting algorithm 
splits a higher level block into four lower level blocks in order to re-estimate disparity when the 
maximum to minimum MAD ratio of the lower level blocks exceeds a single threshold, thus 
obtaining more detailed intermediate views. 

[0020] However, a little change can occur in disparity due to shaking of the camera, changes 
in lighting conditions, and noise components such as dust, even in a still image or low 
complexity image (with less motion). Since blocks in this kind of image may have MAD ratios 
near a threshold, the determination of whether these blocks are split is highly dependant on 
such changes in disparity. Thus, disparity estimates suffer from a significant change when 
synthesizing intermediate views for such an image block. In this case, since the conventional 
block splitting algorithm applies each single threshold when splitting an LMB into four MSBs and 
an MSB into four SSBs, flickering can occur in synthesized intermediate views. 

[0021] Furthermore, the conventional block splitting algorithm in the quadtree disparity 
estimation approach cannot prevent degradation in quality of intermediate views, in particular, 
flickering near edges, by adopting a single threshold. 

SUMMARY OF THE INVENTION 

[0022] The present invention provides a method and apparatus to prevent flickering when 
synthesizing intermediate views that are required to represent a three dimensional (3D) image. 

[0023] Accordingly, it is an aspect of the present invention to provide a new method of 
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splitting a macro image block (higher level block) into sub blocks (lower level blocks) according 
to a quadtree disparity estimation approach. 

[0024] It is another aspect of the present invention to provide a new apparatus to split a 
macro image block (higher level block) into sub blocks (lower level blocks) according to a 
quadtree disparity estimation approach. 

[0025] Additional aspects and/or advantages of the invention will be set forth in part in the 
description which follows and, in part, will be obvious from the description, or may be learned by 
practice of the invention. 

[0026] The foregoing and/or other aspects of the present invention are achieved by providing 
a method of splitting each of macro image blocks (hereinafter called "macro blocks") in left-eye 
and right-eye views into sub image blocks ("sub blocks") according to quadtree disparity 
estimation, and then splitting each sub block into smaller sub blocks. The method includes: 
setting a plurality of splitting threshold values for a macro block in an image frame and 
determining whether to split the macro block into the sub blocks; and setting a plurality of 
splitting threshold values for each sub block and determining whether to split each sub block 
into smaller sub blocks. 

[0027] The foregoing and/or other aspects of the present invention may also be achieved by 
providing a method of splitting an image block the method comprising: setting a plurality of 
macro block splitting threshold values to split a macro block in an image frame into sub blocks 
and determining whether to split the macro block according to whether a macro block at the 
same location in a preceding image frame as the current macro block (hereinafter called 
"preceding macro block") has been split; and setting a plurality of sub block splitting threshold 
value for splitting the sub block into smaller sub blocks and determining whether to split each 
sub block into smaller sub blocks according to whether a sub block at the same location in the 
preceding macro block as the current sub block (hereinafter called "preceding sub block") has 
been split. 

[0028] The foregoing and/or other aspects of the present invention may also be achieved by 
providing an apparatus to split an image block, the apparatus comprising: a macro block 
splitting determining unit that sets a plurality of macro block splitting threshold values to split a 
macro block in an image frame into sub blocks and determines whether to split the macro block; 
and a sub block splitting determining unit that sets a plurality of sub block splitting threshold 
values to split each sub block into smaller sub blocks and determines whether to split each sub 
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block. 

[0029] The foregoing and/or other aspects of the present invention may also be achieved by 
providing an apparatus to split an image block, the apparatus comprising: a macro block 
splitting determining unit that sets a plurality of macro block splitting threshold values to split a 
macro block in an image frame into sub blocks and determines whether to split the macro block 
according to whether a macro block at the same location in a preceding image frame as the 
current macro block (hereinafter called "preceding macro block") has been split; and a sub block 
splitting determining unit that sets a plurality of sub block splitting threshold values to split each 
sub block into smaller sub blocks and determines whether to split each sub block according to 
whether a sub block at the same location in a preceding macro block as the current sub block 
(hereinafter called "preceding sub block") has been split. 

[0030] The foregoing and/or other aspects of the present invention may also be achieved by 
providing a recording medium on which a method is written as a program code that can be read 
and executed on a computer, the program coded method comprising: setting a plurality of 
splitting threshold value for a macro block in an image frame and determining whether to split 
the macro block into sub blocks; and setting a plurality of splitting threshold value for each sub 
block and determining whether to split each sub block into smaller sub blocks. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0031] These and/or other aspects and advantages of the present invention will become 
apparent and more readily appreciated from the following description of the embodiments, taken 
in conjunction with the accompanying drawings of which: 

[0032] FIG. 1 A is a flowchart showing a conventional image block splitting method; 

[0033] FIG. 1B shows image blocks produced in a block splitting procedure according to a 
quadtree disparity estimation approach; 

[0034] FIG. 2A is a flowchart showing a block splitting method according to an embodiment of 
the present invention; 

[0035] FIG 2B is a flowchart showing operation S20 of the method of FIG. 2A according to 
another embodiment of the present invention; 

[0036] FIG 2C is a flowchart showing operation S21 of the method of FIG. 2A according to 
another embodiment of the present invention; 

[0037] FIG. 3A is a diagram showing a configuration of an image block splitting apparatus 
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according to another embodiment of the present invention; 

[0038] FIG 3B is a diagram showing the configuration of an LMB splitting determining unit 
shown in FIG. 3A according to another embodiment of the present invention; and 

[0039] FIG 3C is a diagram showing the configuration of the MSB splitting determining unit 
shown in FIG 3A according to another embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0040] Reference will now be made in detail to the embodiments of the present invention, 
examples of which are illustrated in the accompanying drawings, wherein like reference 
numerals refer to the like elements throughout. The embodiments are described below in order 
to explain the present invention by referring to the figures. 

[0041] Referring to FIGS. 2A-2C, in operation S20, a plurality of splitting threshold values 
are set for a first NxN large macro block (LMB) to be processed, and then splitting is determined 
depending on the set splitting thresholds. Hereinafter, to aid in understanding, it is assumed 
that N is 16 (generally, a macro block size is 16x16 or8x8). 

[0042] In operation 201 , splitting threshold values are set for a 16x16 LMB by experiment. It 
is an aspect of the present invention to set two threshold values; a threshold value Rthui to 
determine the possibility of splitting the LMB, and a threshold value Rthsi to determine whether 
to split the LMB. Since block splitting is performed to ensure the accuracy of estimation, the 
threshold values are preferably set based on mean absolute difference (MAD) ratios used in 
disparity estimation as described above. In this invention, MAD ratios calculated when splitting 
LMBs into MSBs and when splitting MSBs into SSBs are denoted by Rmadsubi and R ma dsub2, 
respectively. 

[0043] Here, it is an aspect to determine whether to split the 16x16 LMB (operation S20) by 
referring to whether the preceding 16x16 LMB has been split. 

[0044] It is another aspect to determine whether to split the LMB by using only the above 
threshold values without referring to the result of processing the preceding 16x16 LMB. For 
example, while an LMB is not split if R ma dsubi approximates Rth U i, the LMB may be split if Rmadsubi 
approximates R^si- That is, whether to split the LMB can be determined by setting a 
predetermined reference value for Rmadsubi- 

[0045] Nevertheless, it is necessary to refer to splitting of the preceding 16x16 LMB due to 
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ambiguity in the setting of the predetermined reference values and information about frames 
(blocks) in image data being similar. Thus, the image data may have hysteresis characteristics 
according to which processing of the current frame (block) is affected by the result of processing 
of the preceding frame (block). This invention determines whether to split an LMB to be 
processed using these characteristics of image data. The importance and reason for referring 
to splitting of the preceding LMB will be more clearly described infra. 

[0046] Operation S20 is divided into operations S202 and S203. In operation S202, whether 
or not to split the LMB is determined by deciding whether Rmadsubi is greater than R mu1 . Here, 
Rmadsubi denotes the ratio of maximum MAD to minimum MAD of an 8x8 MSB within the 16x16 
LMB. If Rmadsubi is greater than R^ui, in operation 203, Rmadsubi, Rthui, and Rthsi are compared 
with one another to determine whether to split the LMB. 

[0047] After the threshold values R^ui and R^si of the LMB are set in the operation S201 , the 
operation S202 is performed to determine if Rmadsubi is less than Rthui- If Rmadsubi is less than 
Rthui, which means that disparity has been correctly estimated (there is no possibility of splitting 
the LMB), it is finally decided not to split the LMB. 

[0048] Conversely, if R ma dsubi is greater than R thu1 in the operation S202, which means there 
is a possibility of splitting the LMB, operation S203 is performed to determine whether to split 
the LMB. Unlike conventional techniques, the present invention does not directly target the 
LMB for splitting even when it is determined that there is no possibility of splitting it. Instead, the 
present invention further includes the operation S203 of determining whether to split the LMB. 

[0049] When determining whether to split the LMB in the operation S203, first, it is 
determined in operation S2031 whether Rmadsubi is between R^ui and Rthsi, which is the 
threshold value to determine whether to split the LMB. If Rmadsubi is not between R^ui and R^si, 
which means Rmadsubi is greater than R^si, it is determined to split the LMB and the LMB is split 
into MSBs. Then, the process proceeds to operation S21 of determining whether to split the 
MSBs. 

[0050] If Rmadsubi is between R mu1 and R^si, which means that the LMB will be split or there is 
a possibility of splitting the LMB, to ensure accuracy in determination of the splitting possibility, 
the present invention further includes operation S2032 of determining whether an LMB placed at 
the same location in the preceding image frame as the LMB to be processed (hereinafter called 
"preceding LMB") has been split. Thus, the LMB to be processed is split, or whether to split the 
LMB is determined by referring to whether the preceding LMB has been split. 
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[0051] As described supra, the reason for referring to whether the preceding LMB has been 
split becomes evident through operation S203. In particular, it is necessary to refer to splitting 
of the preceding LMB when the determination of whether to split the current LMB is ambiguous. 
That is, if Rmadsubi is between R mu i and R^, a minute change in disparity may occur, as 
described in the conventional technique. In such an image block, since the MAD ratio is near 
the splitting threshold, there is often ambiguity as to whether to split. If intermediate views are 
synthesized by unconditionally either splitting or not splitting the image block, the intermediate 
views may exhibit flickering. Thus, the present invention assures accuracy in the determination 
of whether to split by referring to whether the preceding LMB has been split. 

[0052] When R ma dsubi is between Rthui and Rthsi and it is ambiguous whether to split the 
current LMB, it is determined that the current LMB is to be split if the preceding LMB is found to 
have been split. If the preceding LMB has not been split, it is determined that the current LMB 
is not to be split. When R ma dsubi is between R^ui and R^si and the current LMB is split without 
referring to whether the preceding LMB has been split, unnecessary splitting may degrade the 
quality of intermediate views synthesized after splitting. Thus, unnecessary splitting can be 
prevented by referring to whether the preceding LMB has been split. 

[0053] In order to refer to the splitting of the preceding LMB, a system implemented by this 
invention is required to have a separate information storage unit to store the result of splitting 
the preceding LMB. For example, the processing result may be stored in tabular form as shown 
in Table 1 below. Table 1 shows splitting information about the preceding block when an image 
of 720x288 pixels per frame is split into 810 macro blocks (MSBs) of 16x16 pixels. 



<Table 1> 



Macro block 


Splitting information about preceding block 




LMB 


MSB1 


MSB2 


MSB3 


MSB4 


! 1 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 














809 


1 


0 


0 


0 


1 


810 


0 


0 


0 


0 


0 



[0054] In Table 1 , c 0' means the preceding LMB has not been split while '1' means it has been 
split. MSB1-MSB4 denote sub blocks in each LMB, and splitting information about the 
preceding block described above is updated with the result of processing the current LMB when 
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starting the procedure to determine whether to split the next LMB. 

[0055] If it is determined whether to split the LMB, then the process proceeds to operation 
S21 of determining whether to split the resulting MSBs. 

[0056] In operation S21, a plurality of splitting threshold values are set for sub blocks, and 
then it is determined whether to split each sub block into smaller sub blocks. Specifically, a 
plurality of splitting threshold values are set for an 8x8 sub block (MSB) into which the first 
16x16 macro block was split in operation S20, and then it is determined whether to split each 
MSB according to the splitting threshold values. 

[0057] Here, splitting threshold values of MSBs are set by experiment, like the LMB splitting 
thresholds. It is also preferable to set two threshold values: a threshold Rth U 2 to determine the 
possibility of splitting an MSB and a threshold Rth S 2 to determine whether to split an MSB. Since 
block splitting is performed to ensure the accuracy of estimation, the threshold values may be 
set based on MAD ratios used in disparity estimation, as in the case of splitting an LMB. 

[0058] It is more preferable that whether to split an 8x8 MSB (operation S21 ) is determined 
by referring to whether the preceding 8x8 MSB has been split, as in the case of LMB splitting. 
The background and reason for referring to whether the preceding 8x8 MSB has been split are 
the same as in the case of LMB splitting, so a detailed description thereof will be omitted. 

[0059] Operation S21 is divided into operations S212 and S213. In operation S212, the 
possibility of splitting an MSB is determined by deciding whether R mac isub2 is greater than R thU 2. 
Here, R ma dsub2 denotes the ratio of maximum MAD to minimum MAD of a smaller 4x4 sub block 
(SSB) within the 8x8 MSB. If R ma dsub2 is greater than R thU 2, in the operation 213, R ma dsub2, Rthu2, 
and Rths2 are compared with one another to determine whether to split the MSB. 

[0060] After the threshold values Rth u2 and Rths2 of MSB are set in operation S211 , operation 
S212 is performed to determine if R m adsub2 is less than Rthu2- If Rmadsub2 is less than Rthu2, which 
means that disparity has been correctly estimated (there is no possibility of splitting the MSB), it 
is finally decided that the MSB is not to be split. 

[0061] Conversely, if R ma dsub2 is greater than Rth U 2 in operation S212, which means there is a 
possibility of splitting the MSB, operation S213 is performed to determine whether to split the 
MSB. Unlike conventional techniques, the present invention further involves step S213 of 
determining whether to split the MSB, instead of directly targeting the MSB to split even when it 
is determined that the MSB has no possibility of being split. 



10 



[0062] When determining whether to split the MSB in operation S213, first, it is determined in 
operation S2131 whether R ma dsub2 is between Rth U 2 and Rths2, which is the threshold value to 
determine the splitting of MSB. If R ma dsub2 is not between Rth U 2 and R^, which means R m adsub2 is 
greater than R^, it is determined to split the MSB and the MSB is split into four SSBs. Then, 
the process proceeds to the operation of determining whether to split the next MSB. 

[0063] If 

Rmadsub2 is between Rttiu2 and Rths2» which means there is a possibility of splitting the 
MSB, to ensure accuracy in determination of the splitting possibility, the present invention 
further includes operation S2132 of determining whether an MSB placed at the same location in 
the preceding LMB as the current MSB (hereinafter called "preceding MSB") has been split, as 
in the case of splitting LMBs. Thus, splitting of the MSB currently being processed is 
determined by referring to whether the preceding MSB has been split. 

[0064] As described above, the reason for referring to whether the preceding MSB was split 
becomes evident through operation S21 3. In particular, it is necessary to refer to splitting of the 
preceding MSB when the determination of whether to split the MSB currently being processed is 
ambiguous. That is, if Rmadsub2 is between Rthu2 and Rth$2» a minute change in disparity may 
occur as described in the conventional technique. Since MAD ratio is near a splitting threshold 
in such an image block, an unambiguous determination of whether to split often cannot be 
made. If intermediate views are synthesized by splitting or not splitting the image block without 
referring to the result of processing the preceding block, the intermediate views exhibit 
flickering. Thus, the present invention ensures a reliable determination of whether to split by 
referring to whether the preceding MSB has been split. 

[0065] When R ma dsub2 is between Rthu2 and Rths2, and the determination of whether to split the 
current MSB is ambiguous, it is determined to split the current MSB if the preceding MSB is 
found to have been split. If the preceding MSB has not been split, it is determined to not split 
the current MSB. When R ma dsub2 is between Rth U 2 and Rths2 and the current MSB is split without 
referring to whether the preceding MSB has been split, unnecessary splitting may degrade the 
quality of intermediate views synthesized after splitting. Thus, unnecessary splitting can be 
prevented by referring to whether the preceding MSB has been split. 

[0066] In order to refer to the splitting of the preceding MSB, a system implemented by this 
invention is required to have a separate information storage unit to store the result of splitting 
the preceding MSB. For example, the processing result may be stored in tabular form as shown 
in Table 1 above. 
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[0067] Splitting information about the preceding block described in Table 1 is updated with 
the result of processing the current MSB when starting the procedure to determine whether to 
split the next MSB. 

[0068] If it is determined to split the MSB, the procedure to determine whether to split the 
MSB is finished by splitting the MSB into four SSBs. Then, the same splitting process proceeds 
in order to determine whether to split the next LMB. 

[0069] FIG. 3A is a diagram showing the configuration of an image block splitting apparatus 
according to an embodiment the present invention, FIG 3B is a diagram showing the 
configuration of an LMB splitting determining unit shown in FIG 3A according to another 
embodiment of the present invention, and FIG 3C is a diagram showing the configuration of an 
MSB splitting determining unit shown in FIG 3A according to another embodiment of the 
present invention. 

[0070] An LMB splitting determining unit 30 sets a plurality of splitting threshold values for a 
first NxN macro block (LMB) to be processed and determines whether to split the LMB 
according to the splitting threshold values. Hereinafter, it is assumed that N is 16, as is 
assumed in the above description of the method according to the present invention. 

[0071] Splitting threshold values are set for a 16x16 LMB by experiment. As in the method 
according to the present invention, it is preferable to set two threshold values: a threshold value 
Rthui to determine the possibility of splitting an LMB and a threshold value Rthsi to determine 
whether to split the LMB. Since block splitting is performed to ensure the accuracy of 
estimation, the threshold values are preferably set based on MAD ratios used in disparity 
estimation. To maintain consistency with the method of the present invention, MAD ratios 
calculated when splitting LMB into MSBs and when splitting MSB into SSBs are denoted by 
Rmadsubi and R ma dsub2, respectively. 

[0072] Here, it is more preferable that whether to split the 16x16 LMB is determined by 
referring to whether the preceding 16x16 LMB has been split. The background and reason for 
this are the same as in the method of the present invention, so a detailed description thereof will 
be omitted here. 

[0073] The LMB splitting determining unit 30 includes of an LMB splitting possibility 
determining portion 301 and an LMB splitting determining portion 302. The LMB splitting 
possibility determining portion 301 determines the possibility of splitting LMB by deciding 
whether R ma dsut>i is greater than R^ui- Here, Rmadsubi denotes the ratio of maximum MAD to 
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minimum MAD of an 8x8 sub block (MSB) within the 16x16 LMB. If R ma dsubi is greater than 
Rthui » the LMB splitting determining portion 302 compares Rmadsubi > Rthui, and Rthsi with one 
another and determines whether to split the LMB. 

[0074] After the threshold values Rthui and of the LMB are set, the LMB splitting 
possibility determining portion 301 determines if R ma dsubi is less than R mu1 . If Rmadsubi is less 
than Rthui, which means that there is no possibility of splitting the LMB, the LMB is not split. 

[0075] Conversely, if the LMB splitting possibility determining portion 301 determines that 
Rmadsubi is greater than R thu i, i.e., that there is a possibility of splitting the LMB, the LMB splitting 
determining portion 302 finally determines whether to split the LMB. First, a preceding LMB 
splitting determiner 3021 determines whether Rmadsubi is between R mu1 and R^si, which is the 
threshold value to determine splitting of the LMB, and then determines whether the preceding 
LMB has been split. If Rmadsubi is not between R^ui and Rth S i, which means Rmadsubi is greater 
than Rthsi, an LMB splitting final determiner 3022 finally determines whether to split the LMB. 

[0076] If the preceding LMB splitting determiner 3021 determines that Rmadsubi is between 
Rthui and R^si, the LMB splitting final determiner 3022 finally determines whether to split the 
LMB being processed by referring to the determination of whether the preceding LMB was split 
made by the preceding LMB splitting determiner 3021. 

[0077] As described above, in particular, it is necessary to refer to splitting of the preceding 
LMB when the determination of whether to split the LMB being processed is ambiguous. That 
is, if Rmadsubi is between R^ui and R^si, a minute change in disparity may occur, as described in 
the conventional technique. Since MAD ratio is near a splitting threshold value in this kind of an 
image block, an unambiguous determination of whether to split often cannot be made. If 
intermediate views are synthesized by splitting or not splitting image blocks without referring to 
whether the preceding block has been split, the intermediate views exhibit flickering. Thus, the 
present invention ensures a reliable determination of whether to split by referring to whether the 
preceding LMB has been split. 

[0078] When Rmadsubi is between R thu i and R^i, the LMB splitting final determiner 3022 finally 
determines that the current LMB will be split if the preceding LMB is found to have been split by 
the preceding LMB splitting determiner 3021 . On the other hand, if the preceding LMB has not 
been split, the LMB splitting final determiner 3022 finally determines the current LMB will not be 
split. When Rmadsubi is between R^ui and R ms i and the current LMB is split without referring to 
whether the preceding LMB has been split, unnecessary splitting may degrade the quality of 
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intermediate views synthesized after splitting. Thus, unnecessary splitting can be prevented by 
referring to whether the preceding LMB has been split. 

[0079] In order to refer to the splitting of the preceding LMB, the apparatus of the present 
invention is required to have a splitting information storage unit 32 to store the result of 
processing the preceding LMB. For example, the processing result may be stored in tabular 
form as shown in Table 1 above. 

[0080] If it is determined whether to split the LMB, then an MSB splitting determining unit 31 
determines whether to split resulting MSBs. 

[0081] The MSB splitting determining unit 31 sets a plurality of MSB splitting threshold 
values, and then determines whether to split each MSB into smaller sub blocks. Specifically, the 
MSB splitting determining unit 31 sets a plurality of splitting threshold values for 8x8 MSBs into 
which the first 16x16 macro block has been split by the LMB splitting determining unit 30, and 
then determines whether to split each MSB according to the splitting threshold values. 

[0082] Here, splitting threshold values of the 8x8 MSB are set by experiment, like LMB 
splitting threshold values. As in the method according to the present invention, it is also 
preferable to set two threshold values: a threshold value Rth U2 to determine the possibility of 
splitting an MSB and a threshold value Rth S 2 to determine whether to split the MSB. Since block 
splitting is performed to ensure the accuracy of estimation, the MSB splitting thresholds may be 
set based on MAD ratios as described above. 

[0083] Here, it is more preferable that whether to split each 8x8 MSB is determined by 
referring to whether the preceding 8x8 MSB has been split. The background and reason for this 
are the same as in LMB splitting described above, so a detailed description thereof will be 
omitted here. 

[0084] The MSB splitting determining unit 31 includes of an MSB splitting possibility 
determining portion 311 and an MSB splitting determining portion 311. The MSB splitting 
possibility determining portion 311 determines the possibility of splitting an MSB by deciding 
whether Rmadsub2 is greater than Rthu2« Here, Rmadsub2 denotes the ratio of maximum MAD to 
minimum MAD of a 4x4 sub block (SSB) within the 8x8 MSB. If Rmadsub2 is greater than Rthu2i 

the 

MSB splitting determining portion 312 compares R m adsub2, Rthu2, and Rth S 2 with one another and 
determines whether to split the MSB. 

[0085] After the threshold values R thU 2 and R mS 2 are set, the MSB splitting possibility 
determining portion 311 determines if R ma dsub2 is less than Rthu2. If Rmadsub2 is less than Rth U 2, 
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which means that there is no possibility of splitting the MSB, the MSB is not split. 

[0086] Conversely, if the MSB splitting possibility determining portion 311 determines that 
Rmadsub2 is greater than Rth U 2, i.e., that there is a possibility of splitting the MSB, the MSB splitting 
determining portion 312 finally determines whether to split the MSB. First, a preceding MSB 
splitting determiner 3121 determines whether R m adsub2 is between Rth U 2 and Rths2, which is the 
threshold value to determine splitting of an MSB, and then determines whether the preceding 
MSB has been split. If R ma dsub2 is not between Rth U 2 and Rths2, which means R ma dsub2 is greater 
than Rths2, an MSB splitting final determiner 3122 finally determines the splitting of the MSB. 

[0087] If the preceding MSB splitting determiner 3121 determines that R ma dsub2 is between 
Rthu2 and Rths2, the MSB splitting final determiner 3122 finally determines whether to split the 
MSB being processed by referring to the determination of whether the preceding MSB was split 
by the preceding MSB splitting determiner 3121. 

[0088] As described above, it is necessary to refer to whether the preceding MSB was split 
when the initial determination of whether to split the MSB being processed is ambiguous. That 
is, if R m adsub2 is between Rthu2 and Rth S 2, a minute change in disparity may occur as described in 
the conventional technique. Since MAD ratio is near a splitting threshold in this kind of an 
image block, a determination of whether to split may be ambiguous. If intermediate views are 
synthesized by splitting or not splitting image blocks without referring to whether the preceding 
block has been split, the intermediate views exhibit flickering. Thus, this invention ensures a 
reliable determination of whether to split by referring to whether the preceding MSB has been 
split. 

[0089] When R ma dsub2 is between R thU 2 and Rth S2) the MSB splitting final determiner 3122 
finally determines that the current MSB will be split if the preceding MSB is found to have been 
split by the preceding MSB splitting determiner 3121. On the other hand, if the preceding MSB 
has not been split, the MSB splitting final determiner 31 22 finally determines that the current 
LMB will not be split. When R ma dsub2 is between Rth U 2 and R mS 2 and the current LMB is split 
without referring to whether the preceding MSB has been split, unnecessary splitting may 
degrade the quality of intermediate views synthesized after splitting. Thus, unnecessary 
splitting can be prevented by referring to whether the preceding MSB has been split. 

[0090] In order to refer to the splitting of the preceding MSB, the apparatus of the present 
invention is required to have the splitting information storage unit 32 to store the result of 
processing the preceding MSB. For example, the processing result may be stored in tabular 
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form as shown in Table 1 above. 

[0091] Splitting information about the preceding block described in Table 1 is updated with 
the result of processing the current MSB when starting the procedure to determine whether to 
split the next LMB. 

[0092] If the MSB is determined to be split, the procedure to determine whether to split the 
MSB is finished by splitting the MSB into four SSBs. Then, the same splitting process proceeds 
in order to determine whether to split the next LMB. 

[0093] The present invention is advantageous over conventional block splitting techniques in 
that it can synthesize intermediate views in such a way as to represent a high-equality three 
dimensional (3D) image, even over a very small change between blocks (frames), by setting two 
thresholds and using splitting information about a block preceding a block currently being 
processed. 

[0094] Embodiments of the present invention can be written in computer-readable code on a 
computer-readable recording medium. Examples of the computer-readable recording medium 
include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage 
device. The code may also be transmitted via carrier waves, e.g., over the Internet. 
Furthermore, the computer-readable code may be stored on and executed from recording 
media distributed among computer systems connected to one other by a network. 

[0095] Although the preferred embodiments of the present invention described above refer 
only to application to intermediate views for 3D image representation and a two-operation 
splitting procedure (LMB— >MSB— >SSB), the present invention may be embodied as a splitting 
procedure containing more than two operations, depending on the size of a macro block, and 
may be applied to general image processing. Therefore, the described embodiments should be 
considered not in terms of restriction but in terms of explanation. The scope of the present 
invention is not limited by the foregoing description but by the following claims, and all 
differences within the range of equivalents thereof should be interpreted as being covered by 
the present invention. 
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